1. The Field of the Invention
The present invention relates to remote direct memory access. More particularly the present invention relates to methods, systems, and computer program products for reducing communication overhead between protocol layers to make remote direct memory access more efficient.
2. Background and Related Art
In computer systems, data frequently must be copied from one buffer to another. Without direct memory access (DMA), copy operations are performed by a computer system's central processing unit (CPU). While performing a copy operation, the CPU is unavailable for more important and more sophisticated operations. Accordingly, depending on the amount of data to copy, use of a CPU in buffer copy operations may lead to poor resource management and result in slow system performance.
In traditional DMA, the CPU programs a DMA engine to perform the buffer copy, such as by specifying a source address, a destination address, and a length to copy. The DMA engine then moves the data from one buffer to another, without further impact on CPU resources. When the buffer copy is complete, the DMA engine notifies the CPU.
FIG. 1 illustrates a remote DMA between host 100A and host 100B. In order to copy the contents of buffer 120A at host 100A to buffer 120B at host 100B, hosts 100A and 100B program remote network interface cards (RNICs) 110A and 110B to make the transfer, such as by identifying a source address at a particular host, a destination address at a particular host, and a length to copy. This frees, or reduces the utilization of, CPUs 130A and 130B at hosts 100A and 100B, making CPUs 130A and 130B more available for other activities. RNIC 110B receives the data and transfers it to buffer 120B, without using CPU 130B. Thus, RNICs 110A and 110B operate as remote DMA (RDMA) engines, with corresponding RMDA interfaces 112A and 112B. RDMA transfers use zero copy paths 124A and 124B, allowing for data transfer (throughput) at the rate of the network or fabric. As in traditional DMA, the RNICs notify the appropriate CPU when finished making the transfer.
Note that in FIG. 1 two additional paths from the RNICs 110A and 110B to buffers 120A and 120B are illustrated. Buffer copy paths 122A and 122B are directed to buffers 120A and 120B through intermediate private buffer pools 140A and 140B, respectively, whereas zero copy paths 124A and 124B directly connect to buffers 120A and 120B. Buffer copy paths 122A and 122B use CPUs 130A and 130B for copy operations. In other words, buffer copy paths 122A and 122B result in higher CPU utilization, meaning the CPU is less available to perform other operations. It also should be noted here that network throughput (data transfer at network speed) is increasingly likely to exceed the transfer rate available from an RNIC's CPU. Even so, one reason for the two separate paths relates to the communication overhead associated with zero copy paths 124A and 124B. Remote direct memory access using zero copy paths 124A and 124B introduces some amount of overhead to RDMA (e.g., the overhead to program RNICs 110A and 110B for an RDMA transfer). As described in more detail with respect to FIGS. 3A and 3B below, for small data transfers, this overhead may negate the faster data throughput that zero copy RDMA offers. In other words, for small data transfers, it may be faster to use the buffer copy path, and sacrifice CPU utilization at the RNIC, rather than use the zero copy path that RDMA offers.
Conventionally, the zero copy path overhead for RDMA has been relatively significant, requiring transfers on the order of 32 k bytes and above to justify its use. As can be appreciated, many networks frequently have data transfers of less than 32 k bytes, making zero copy RDMA less attractive for certain operations. Accordingly, methods, systems and computer program products are desired to reduce the communication overhead for zero copy paths so the performance benefits of faster data throughput can be used for relatively smaller data transfers.